Publication review user interface and system

ABSTRACT

Disclosed are systems, apparatus, and methods for managing user reviews of publications within an online user network. Various user interfaces may provide access to view a publication and request a review thereof, solicit and receive reviewing information from the requesting or another user (or multiple users), and provide access to a review generated from the received reviewing information.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of priority to U.S.Provisional Application Ser. No. 61/953,543, filed Mar. 14, 2014, andalso U.S. Provisional Application Ser. No. 61/967,333, filed Mar. 14,2014, the entirety of each of which is hereby incorporated by referenceherein.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever. The following notice applies to the software and dataas described below and in the drawings that form a part of thisdocument: Copyright 2014, RESEARCHGATE CORPORATION, All Rights Reserved.

TECHNICAL FIELD

This patent document pertains generally to publication technology, andmore particularly, but not by way of limitation, to a user interface andsystem to harvest and publish review information relating to apublication.

BACKGROUND

The automated harvesting and publication of review information (e.g.,related to an academic or technical publication) in an open andtransparent manner may present a number of technical challenges,specifically with respect to the person-machine interface to solicit,receive, and then publish such review information. Free-form reviewsubmission interfaces may not, for example, harvest review informationthat is particularly uniform, or that encourages reviewers (e.g.,scientist or researchers) to focus review comments on any one or moreaspects of a publication that are particularly pertinent. Furthermore,the automatic harvesting and publishing of reviews may, due to the lackof (or reduction in) human oversight, fail to establish a level ofconsumer trust in the review quality that is comparable with thatgenerally placed in traditional human-managed peer reviews (asimplemented, e.g., by many scientific journals). In addition, since theautomated system ultimately depends on human input of reviewinformation, its success and utility to the community of reviewconsumers at large is a function of the ability of the system toencourage or incentivize reviewer participation.

BRIEF DESCRIPTION OF DRAWINGS

Some embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings in which:

FIG. 1 is a block diagram depicting a social-network and publicationsystem with integrated review functionality in accordance with variousembodiments.

FIG. 2A is a schematic user interface diagram illustrating an examplepublication user interface providing information about and access to apublication in accordance with various embodiments.

FIG. 2B is a schematic user interface diagram illustrating an examplepublication user interface providing, along with information about thepublication, access to a review of the publication, in accordance withvarious embodiments.

FIGS. 3A and 3B are a flow chart illustrating a method for harvestingand publishing review information in accordance with variousembodiments.

FIGS. 4A-4C are schematic user interface diagrams illustrating exampleuser interface screens for soliciting and receiving publication reviewinformation from a user in accordance with various embodiments.

FIG. 5 is a schematic user interface diagram illustrating an examplereview user interface providing a review overview in accordance withvarious embodiments.

FIGS. 6A and 6B are schematic user interface diagrams illustrating anexample review user interface providing review information to a consumerthereof in accordance with various embodiments.

FIGS. 7A and 7B provide an entity-relationship diagram depicting variousdata objects storing information used within a social network andpublication system with publication review functionality in accordancewith various embodiments.

FIG. 8 is a block diagram illustrating the relations between variousprogram modules, data objects, and user interfaces in accordance withvarious embodiments.

FIG. 9 is a block diagram illustrating an example of a softwarearchitecture that may be installed on a machine, according to variousembodiments.

FIG. 10 is a block diagram of a machine in the form of a computer systemwithin which a set of instructions for causing the machine to performany one or more of the methodologies discussed herein may be executed,in accordance with various embodiments.

DETAILED DESCRIPTION

Described herein are systems and methods for integrating reviews ofpublications into an online social network, allowing network users topost and/or facilitate access to their own original publications, reviewother users' publications, read and provide feedback on reviews, and/orotherwise engage in an open discourse about publications. Beneficially,in various embodiments, not only the impact and quality of a user'spublications (as determined, e.g., based on the number of views andcitations as well as the reviews), but also the quality of reviews shecontributes (as determined, e.g., based on feedback on the reviews)and/or her overall level of engagement (as determined, e.g., based onthe frequency of postings, whether publications, reviews, comments, orother) can affect her standing and reputation within the social network(implicitly via other users' reaction to her postings, and/or viaexplicit scores, rankings, etc. computed by the automated system),providing a strong incentive for publishing high-quality work andreviews and/or otherwise contributing to the discourse. In someembodiments, publication reviewers receive guidance from the automatedsystem via a set or sequence of structured review questions, enhancingthe uniformity and relevancy of the reviews.

In various embodiments, the social network is closed or partially closedinasmuch as access to certain functionality, such as the ability tocontribute content, is limited to a specified user group (whereas, e.g.,read access may be provided to the public at large). For example, insome embodiments, the network is limited to academic users, and thecontent focusses on academic publications and reviews and communicationsrelated thereto. While the example of an academic publication and reviewsystem is used throughout this disclosure for illustration purposes,many of the embodiments and features described herein may alsoapplicable to other user groups and types of publications. To name but afew examples, systems and methods in accordance herewith may findapplication in the publication, review, and discussion of works ofliterature, film, art, or music; news and other journalisticpublications; meal recipes; etc. It will be readily apparent to those ofordinary skill in the art, from the following detailed description anddrawings, how the embodiments described herein can be adjusted to suchother application contexts.

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of some example embodiments. It will be evident, however,to one skilled in the art that the present invention may be practicedwithout these specific details.

FIG. 1 is a block diagram depicting a social network and publicationsystem 100, according to an example embodiment, that integratesfunctionality for harvesting and publishing review informationpertaining to publications (e.g., textual, graphic, image, video etc.).The social network and publication system 100 may including varioussub-systems such as a publication processing system 102, a reviewprocessing system 104 (which may, but need not, form a component withinthe publication processing system 102), a publication recommendationsystem 106, and a review recommendation system 108, all of whichinteract with a social network presentation system 110. Users 112 (e.g.,users A, B, and C) may register with the social network presentationsystem 110 by providing user-profile information, such as a name,institutional affiliation, contact information (e.g., email address),etc., and establishing network-access credentials such as, e.g., a username (which may be the user's email address) and password. Theuser-profile information may be stored within a user-profile database114 of the social network presentation system 112. In some embodiments,eligibility to register as a user is based on certain conditions, suchas academic credentials, and may be determined based on theuser-provided profile information during sign-up. For example, theability to register may be contingent upon providing an email addressassociated with an academic or research institution recognized by thesystem; a comprehensive list of such institutions may be established,and extended from time to time upon request and/or as necessary, by asystem administrator. Alternative factors that may render a usereligible to register include, for instance, co-authorship of apublication posted by an existing registered user or other evidence ofinvolvement in the creation of scientific knowledge, e.g., in the formof published articles or association with an academic institution or aresearch-and-development function of a governmental or corporateinstitution, as may be proven to a system administrator in anapplication process.

Once registered, a user 112 may have the ability to upload herpublications to the system 100. Alternatively or additionally, thesystem 100 may conduct a batch import of publications, e.g., bydownloading them from openly accessible third-party publicationrepositories 116 (e.g., as provided on the web sites of manyuniversities); such batch-import functionality may be provided by apublication batch data connector 118. Subsequently, the system 100 mayallow its users to link the already downloaded publications to theirrespective user profiles by claiming authorship (or co-authorship). Forinstance, as part of the sign-up process, the system may automaticallypresent the new user 112 with a list of publications that match theuser's name (and optionally his institutional affiliation), or haveotherwise been identified as potentially authored or co-authored by thenew user, and request confirmation from the user that he is, indeed, anauthor of this publication. A “publication,” as used herein, may be awork already published by a third party (i.e., outside thesocial-network environment) (to the extent allowed by copyright law),such as an academic article included in a scientific journal, or a(perhaps preliminary) work first published within the social network andpublication system 100, such as a draft of an academic article that hasnot yet been submitted for publication and/or is not intended forsubmission to any journal. (Sometimes, to comply with the copyrights ofthird-party publishers, users may upload final drafts of their works,but not the final typeset publication as it appeared in the third-partyjournal.) Alternatively to uploading a full-text version of hispublication directly to the publication system, a user 112 may enterbibliographic information and/or provide a link to an external web sitewhere the publication is available (e.g., for purchase). Someembodiments also allow the user to provide an original publication in astructured format (e.g., including separate sections of text, imagefiles, etc.) rather than in a single, full-text file. The publication isgenerally stored in the system 100 in the form of one or morepublication data objects, such as data structures (e.g., tables,records, or entries within a database) and/or data files. For example,in various embodiments, the publication processing system 102 stores,for each uploaded or entered publication, publication metadata, and, ifavailable, a full-text version of the publication, or otherwise a linkthereto. The publication metadata, and optionally links to full-textdocuments, may be stored in a publication database 120, and full-textfiles (e.g., in the form of pdf documents) may be stored in a separatefull-text publication repository 122.

The social network and publication system 100 may provide searchfunctionality that allows users to search for publications of interestbased on, for example, the field of research, the author's name, orspecific citation information. Alternatively or additionally, thepublication recommendation system 106 may automatically provide a listof potential publications of interest based on the user's profile (whichmay include, e.g., a list of her research interests and/or a list of herown publications) and/or other user-specific information (e.g., herprior search and browsing history within the network, the researchinterests of people to whom she is related within the social network).Whether provided in response to a search request or as an independentsystem recommendation, publications may be ranked based on variouscriteria (in addition to their relevancy to the user's request or thematch to the user's interests in general), including, e.g., the recencyof the publications, the reviews they have received, the reputation oftheir authors, the number of times they have been cited, etc.

When a user selects a certain publication for viewing, she may bepresented with a user interface that provides access to the publicationor a portion thereof; an example publication user interface is shown inFIG. 2A. The user interface may provide access to the publication bydirectly displaying the publication (or portion thereof), or—asillustrated in FIG. 2A—by displaying at least the publication metadata200 (such as bibliographic information including, e.g., the title,authors, journal, and/or an abstract) and providing a link 202 to thefull-text version of the publication (e.g., a link to a system-internalwebsite or to a third-party website from which the publication may bedownloaded) or a user-interface element (e.g., button) allowing the userto request such a full-text version (e.g., by automatically sending amessage to the author indicating the requesting user's interest in thepublication). The user interface may also include links to the authors'user profiles (to the extent they are registered system usersthemselves) or automatically generated author profiles fornon-registered authors, lists 204 of and links to references that citethe publication or were cited in the publication, and/or otherinformation related to the publication. Further, the user interface mayinclude one or more suitable user-interface elements (e.g., one or moresuitably labelled buttons 206, 208 (e.g., “Review this publication”and/or “Review request”)) that allow a user to indicate that she wishesto review the publication or to request a review of the publication byanother user. By activating such a user-interface element 206 or 208,the user may execute a link that leads her to a review-input userinterface, where she may enter or upload review information (such as,e.g., ratings for a number of review questions, free-text comments,supporting documents, relevant citations, etc.). Alternatively tonavigating away from the original publication user interface to anentirely new user interface, activating the user-interface element torequest a review may trigger changes to just a portion of thepublication user interface (e.g., appearance of a comment field in aportion of the screen), the display of pop-up windows where the reviewinformation may be entered, and the like.

The review processing system 104 may guide the user through the reviewby soliciting and receiving review information, e.g., with a sequence ofreview-input screens presented as part of the review-input userinterface. The review processing system 104 further processes thereceived review information for storage, in the form of one or morereview data objects (including data structures and/or data files), in areview data repository 124 (which may include, e.g., one or moredatabases and/or file repositories, and, although depicted separatelyfrom the publication database 120 and full-text repository 122, may beintegrated therewith in whole or in part). For example, the answers to aseries of structured review questions may be stored in a database, andany supporting documents (e.g., images or research data sets) may bestored in a separate repository and properly linked to the databaseentry for the review. The review information is stored in associationwith the publication to which it pertains, e.g., by virtue of suitable(cross-)references between the publication data objects and review dataobjects as integrated in the data objects themselves or stored in aseparate reference index.

Once the publication has been reviewed, the review(s) (or portionsthereof) may be accessible from the user interface for the publicationto which they pertain. For example, links to and/or a summary of thereview(s) may be displayed along with the other publication information.In some embodiments, illustrated in the example user interface shown inFIG. 2B, the publication user interface merely indicates theavailability and/or number of reviews (generally indicated at 210)optionally categorized based on their main conclusion (e.g., whether thepublished research is reproducible), and provides a link to a separatereview user interface where more detailed review information isdisplayed. The review user interface (or the publication user interfaceif displaying review information) may also provide functionality tousers reading the reviews to provide feedback, e.g., by rating thehelpfulness of a review or providing free-text comments on the reviews.This feedback may be processed by the review processing system 104 (or aseparate feedback processing module) and stored, as one or more feedbackdata objects in association with the review, in the review datarepository 124 (or a separate feedback data repository). The feedbackmay be fed into the review recommendation system 108, which may rank thereviews and/or selects reviews for display based on their relativeutility to users. Feedback on reviews may also be used within the socialnetwork and publication system to compute user scores or ratings.Further, as the reader will readily appreciate, feedback on reviews can,of course, implicitly affect the user's reputation and standing withinthe social network, regardless of any explicitly computed and publishedmetrics.

The social network and publication system 100 may include functionalityfor finding and analysing connections and the level of relatednessbetween users (i.e., their “social relatedness,” which may reflect, forexample, direct social connections between two people, affiliation withthe same institution, publications by two people with a commonco-author, receipt of funding from the same source, etc.) and betweenusers, publications, and reviews (i.e., “interest relatedness,” whichmay reflect, for example, a cluster of topics of interest to each user,as may be computed from the user's own publications). Both thepublication recommendation system 106 and the review recommendationsystem 108 may take social and interest relatedness into account inmaking publication and review recommendations, respectively. Further,both sub-systems 106, 108 may consider relationships between publicationauthors, reviewers, and review consumers providing feedback to determinepossible positive or negative biases and adjust the impact of the reviewor feedback based thereon. For example, positive reviews or feedback maybe discounted if originating from co-workers, as determined based oninstitutional affiliations, co-authorship of previous publications, etc.

In various embodiments, every item of user-generated content availableon the system is accompanied by information identifying thecontent-generating user and, optionally, also information indicative ofthe user's reputation. For example, the content-generating user's nameand photo, and optionally a link to her user profile, may be displayedalong with the content. As indicators of the user's reputation, herinstitutional affiliation and/or scores relevant to reputation (such asa score computed by the social network and publication system based ondata known to the system (e.g., on the user's publication record,reviews and ratings by other users, her level of activity within thesystem, etc.), or the well-known impact factor calculated from thenumber of articles a user has published in conjunction with thereputation of the journals in which they were published) may be shown.In this manner, consumers of user-generated content (includingpublications, publication reviews, or comments/feedback on such reviews)can quickly ascertain the reputation of the content-generating user, andtake this information into account when deciding which content to read.

Referring to FIGS. 3A and 3B, an example process 300 for gathering andprocessing review information, and publishing the resulting review,within an online social network and publication system (such as system100 of FIG. 1) will now be described. Starting point of the process isusually that a user is provided with digital access to a certainpublication (or portion thereof) via a publication user interfacegenerated based on one or more publication data objects stored for thepublication (operation 302). For example, as discussed above, thepublication user interface may directly display (part of) thepublication, or display publication metadata and a link to the full-textpublication document. The review process is triggered when the userrequests a review of the publication (which may be a request for theopportunity to review the publication herself, or a request for reviewby another user), e.g., by activating a user-interface element included,for this purpose, in the publication user interface, and the systemreceives this digital user-initiated review request (operation 304).

The review request can indicate the user's desire to review thepublication himself, or his interest in having one or more other usersreview the publication (as determined at 306). In some embodiments, twoseparate user-interface elements are provided on the publication userinterface to implicitly distinguish between these two cases (in whichcase 306 does not involve a separate step). In other embodiments, useractivation of a general review-request user-interface element initiatesa user-system dialog to determine whether the user wishes to provide thereview herself or request another user's review. In yet otherembodiments, the system, by default, proceeds to present thereview-requesting user with a user interface where she can input reviewinformation, but which allows her to indicate that she does not wish toconduct the request review herself In cases where the user requestsreview of the publication by another user, the system (e.g., via itsreview processing system or another suitable sub-system) may identifysuitable potential reviewers (such as researchers in the field to whichthe publication pertains, in particular those with high academiccredentials and/or numerous and/or impactful publications on relatedtopics), and invite the identified candidates to submit a review, e.g.,by sending emails that contain a link to a user interface where reviewinformation can be provided (operation 308). In some embodiments, one ormore suitable reviewers are identified by the system based at least inpart on a match between their interests to the topic of the publicationat issue, where users' interests can be determined, in a simpleimplementation, from user profile information or, in more sophisticatedimplementations, from an interest-relatedness score (as computed, e.g.,based on an analysis of each user's publication record and/or linksbetween users, publications, reviews, etc.).

Once a reviewer request the opportunity to review a publication, oraccepts an invitation to do so (operation 310), the system may solicitand receive review information from her via a review-input userinterface (operation 312), and store the review information in one ormore review data objects associated with the publication (operation314). In various embodiments, the review-input user interface (which mayinclude multiple sequentially displayed screens) is configured to guidethe user through the review process. For example, as depicted, the usermay be presented with a set of structured, predetermined questions (seesub-operation 316) (e.g., relating to the reproducibility of anexperiment, the appropriateness of the methodology or analysis applied,the inclusion of relevant citations to related work, the novelty and/orimportance of the findings, the justification of the conclusion in viewof the results). Associated with each of these questions may be apredetermined set of answers (e.g., yes, no, partly, not applicable),from which the reviewing user may select one answer in response to thequestion (e.g., using radio buttons, a drop-down menu, or some otheruser-interface control element provided in conjunction with the answerchoices) (see sub-operation 318). Alternatively or additionally, theuser may have an opportunity to provide a free-style textual response,e.g., via a text input field (see sub-operation 320). For example, inone embodiment, the text input field may be presented in response toreceiving user selection of an answer from a predetermined set ofanswers. The text input field may facilitate high-quality review byprompting the reviewing user to provide a more thorough response, andalso to justify the selection of the answer from the predetermined set.To allow the reviewing user to conveniently format provided textualreview information, the text input field may provide various textformatting capabilities (e.g., bold, italics, strikethrough, bulletpoints). The reviewing user's answers (whether provided by selection ofone of a set of predetermined answer options and/or in the form of freetext) may be stored in association with the question, e.g., in a reviewdatabase.

In some embodiments, the set of questions is presented to the reviewinguser in a sequential manner, prompting the user for the progressivedisclosure of answers. For example, the user may first be presented witha primary question and a set of reply options, and only upon receipt ofan answer to the primary question are follow-up questions displayed ormade available for answer. Referring to the review-input user-interfacescreens shown in FIGS. 4A-4C, the primary question 400 may, forinstance, ask whether research presented in the publication isreproducible (FIG. 4A). Once the user selects one of the answer options402 (e.g., yes, no, not applicable), a set of follow-up questions 404may be activated, presented, or made available for answer, eithersimultaneously or sequentially. In the illustrated example, fivefollow-up questions related to conclusions, findings, references,analyses, and methodology of the publication being reviewed arepresented, each in conjunction with a user-selectable answer (e.g., yes,no, partly) (FIG. 4B). (In some embodiments, the follow-up questions aredisplayed along with the primary question at the outset, but are notavailable for user selection until the primary question has beenanswered (in other words, they are “greyed out”). This is illustrated inFIG. 4A with shading of the area showing the follow-up questions.) Uponselection of one of the answer choices for any of the questions, atext-entry field 406 may appear, allowing the user to provide a commentsupporting his answer (FIG. 4C). In some embodiments, as shown, thereviewing user is asked to answer at least one of the follow-upquestions. In other embodiments, she is sequentially presented with allof the questions, for instance, such that, when the reviewing user hasprovided an answer to the primary question (e.g., is the researchreproducible), a first follow-up question (e.g., are the interpretationsor conclusions justified by the results?) is visually highlighted andthe predetermined set of answers is enabled for user selection. Notethat, in this example, the other follow-up questions are not enabled foranswer provision until a current follow-up question has received ananswer. In this way, the method walks the reviewing user sequentiallythrough the predetermined set of follow-up questions, enabling asubsequent question for answer upon receiving an answer for a currentquestion.

Referring again to FIGS. 3A and 3B, in some embodiments, a text inputfield facilitating free-text responses to review questions may alsoenable the convenient referencing of certain pages or portions of thepublication being reviewed in support or justification of the reviewer'sanswer. To this end, an “insert page reference” interface component maybe displayed (sub-operation 322), which allows the reviewing user tosearch for and manually select one or more portions of content (e.g.,text or otherwise) from the publication for insertion into the textinput field. The “insert page reference” interface component mayautomatically supplement the reviewer's comments with a reference to theselected portions of content, indicating an exact location within thepublication where the selected portion of content resides; in someembodiments, this reference takes the form of an executable link to therelevant portion of the full-text publication. The inclusion of suchreference location data makes it convenient for a further user readingthe review to validate or review justification information provided bythe reviewing user. Conversely, the “insert page reference” interfacecomponent may also automatically insert a reference in the full-textpublication to the review (operation 324), e.g., by supplementing theportions of content on which a certain review response is based with alink to that response. This allows a reader of the full-text publicationto easily discover relevant review content based on annotations in thetext being read.

In some embodiments, the full-text publication can also be annotateddirectly, and making such an annotation may (but need not in everyembodiment) result in the automatic initiation of a review process.Annotations in a publication may indicate other scientific publicationsin which the annotated publication is cited or which are relevant to thesubject matter of the annotated publication, include links to relevantreviews and discussions, and/or simply record the reader's comment onthe publication. In this manner, a publication becomes a dynamic or“living” document containing the original full-text version as well asvarious discussions, citations, and reviews that arise in connection toit (and/or references thereto).

In some embodiments, a reviewing user further has the option to providereview information beyond answer selections or text responses to thevarious questions, such as by uploading supporting documents. Forexample, a reviewing user who tried to reproduce experimental resultsdiscussed in a publication may be able to upload his data to support hiscontention of the reproducibility or irreproducibility of the publishedresearch being reviewed; following receipt of such supporting data bythe system (operation 326), the supporting data may be associated withthe reviewing user's own profile, e.g., as a published dataset(operation 328). Or, if a reviewer takes the published experimental forgranted, but questions the adequacy of the interpretation or analysis,she may provide her own analysis of the data. Reviewers may also uploadother relevant publications that they think could or should have beencited in the publication being reviewed. In various embodiments, anykind of files can be attached both to the individual review questions orcategories, as well as to the review as a whole; these files may beadded to the reviewer's profile.

In general, review data objects may be associated with the reviewinguser's profile, e.g., such that, upon viewing the reviewer's profile, alist of reviews she submitted appears. In some embodiments, a reviewercan choose to identify other people that contributed to a particularreview (such as co-workers that helped conduct experiments trying toreproduce the results of a publication) (see sub-operation 330); this issimilar to the practice of giving credit to all contributors to ascientific publication. The system may create links between a review andthe user profiles of all contributors to the review. Further, when thereview information is subsequently displayed to a review consumer, thereview contributors may, like the main reviewer, be listed along withthe review.

In some embodiments, review information is received from multiple(independently) reviewing users (generally indicated at 332). In thiscase, the review information may be aggregated across the multiple userreviews (operation 334). For example, from the reviewers' selectionsamong a small number of predetermined answer choices for a set ofquestions (such as yes, no, or partly), the number of times each answerchoice was selected can be determined. Similarly, if, in anotherembodiment, reviewers were asked to rate certain aspects of thepublication on a numerical scale (e.g., from 1-5), averages of theresponses can be computed. The aggregate information may be stored as areview overview data object in the review repository.

Once at least one user has completed her review of a publication and theentered review information has been submitted and stored in the form ofone or more review data objects, the stored information may be processedfor visual presentation in a user interface. For example, the selectedanswers from among the predetermined answer choices may be presented ina compact form, with links to the supporting textual responses and/orother justifying information. In cases where review information has beenreceived from multiple reviewers, a review overview may be generatedfrom the aggregated information. In some embodiments, this overview iscreated regardless of the number of reviews received for a givenpublication. (In this case, the numbers of answers to each qualitativequestion simply add up to one, and the average response to aquantitative question is simply the one response received for thequestion.)

The review overview and/or the individual reviews may be presented inassociation with the publication. To that end, the publication userinterface may be updated to provide digital viewing access to the reviewinformation (operation 336) in one of various alternative manners. Insome embodiments, the review overview and/or individual reviews aredisplayed directly alongside the publication (sub-operation 338). Inother embodiments, the publication user interface displays a reviewoverview, and includes links to the individual reviews (sub-operation340); upon execution of one of the review links, the selected review isthen displayed in a separate review user interface (sub-operation 342).In still further embodiments, the publication user interface merelydisplays a link to the review overview and/or an individual review(sub-operation 344), and the review information itself is shown in aseparate review user interface (sub-operation 346).

In various embodiments, the user interface presenting the review(whether it be the publication user interface or a separate review userinterface) includes functionality for receiving feedback from viewers,or “consumers,” of the review(s) (operation 348). For example, users maybe able to select whether they found a review helpful or not (or ratethe reviews helpfulness on a numerical scale). In addition, they mayhave the opportunity to write comments pertaining to the review, whichmay thereafter be displayed along with the review. The feedback providedmay be used to adjust the presentation of the reviews (operation 350).In addition to supplementing an individual review directly by displayingthe feedback, the system may, for instance, sort or rank reviews for aparticular publication based on the ratings they received, e.g., suchthat the review that is deemed the most helpful appears first. In someembodiments, commenters, like reviewers, have the opportunity to uploadfiles to support the statements they make; these files may be added(e.g., as dataset publications) to the commenter's profile. Furthermore,readers may have the ability to mark a comment as either “helpful” or“not helpful.” This creates another feedback loop that allowsresearchers to curate reviews and ultimately separate the usefulcontributions from biased or otherwise flawed contributions.

In some embodiments, all reviews can be commented on by any user of theplatform. Reviews may be publically visible to any Internet user,whereas interactions (such as commenting on or rating reviews) may belimited to users signed up to the social network. Reviewers and authorsof the publication being reviewed may be specifically identified as such(e.g., on the user interface where the comment or feedback, or accessthereto, is provided) to distinguish them from general commenters. Insome embodiments, comments cannot be made anonymously, to ensure thequality and accountability of the contributor.

In one example embodiment, reviews and feedback thereon are used tocalculate two kinds of “signals” for each publication being reviewed aswell as for each individual review: one indicative of general interestin or the popularity of the subject of the publication or review, andone on the general opinion about the publication or review. Forpublications, general interest can be measured based, at least in part,on the number of reviews the publication has received, and the generalopinion is characterized by the general slant of the reviews (e.g.,favorable, neutral, or unfavorable). Similarly, for individual reviews,general interest can be measured based, at least in part, on the numberof comments and/or the number of feedback votes (regardless whetherfavorable or unfavorable) the review has received, and the generalopinion is reflected in the trend of the feedback, e.g., whether themajority finds the review helpful (or not helpful) or whether opinionsare tied between the two. Scores for the general interest in and generalopinion on a given publication or review may be propagated to theindividual users whose work is being evaluated, i.e., the authors of apublication (providing an assessment as to the popularity of their workand the favorability/unfavorability of reviews) or the reviewers(providing an indication how popular each reviewer's reviews, aggregatedacross publications, are relative to other reviews, and whether theyreceive predominantly positive or negative feedback). Further decisionswithin the system may be based on the scores as propagated to the users.For example, the contents provided by users who receive largely positivereviews and feedback may be more frequently or more prominentlydisplayed or suggested to interested users. Conversely, reviewing userswhose reviews are largely considered unhelpful may be restricted intheir ability to submit further reviews.

FIG. 5 shows an example user-interface screen displaying a reviewoverview. Herein, publication information 500 for the reviewedpublication, such as bibliographic information and an image of thepublication (e.g., an image of the first page, or a selected figure fromthe publication), which may function as an executable link leading tothe full-text version, are displayed at the top. Links to request a(further) review may also be provided. Underneath, a list 502 ofreviewers may be depicted, with links to their individual reviews and,optionally, a very brief indication of the major conclusion of theirreview (e.g., whether the published research is reproducible or not).The reviewers may be identified by name, institution, and optionally aphoto, and one of these components (e.g., the name) may serve as anexecutable link to the reviewing user's profile. Additionally, one ormore scores indicative of the reviewer's reputation may be displayed.

Furthermore, an overview 504 of the aggregated review information may beprovided. This overview allows review consumers to quickly gauge a givenpublication's general perception by the research community. It may showthe spread of the individual answers to the structured questions (e.g.,in the form of a chart that lists, along with each of the questions, thenumber of times each of the possible responses was selected), therebygiving a balanced overview of the publication's perceived merits or lackthereof in the different categories, before a reader has to delve intothe more detailed explanations of individual reviewers in the free textsections. The review overview may include a link to the more detailedindividual reviews. Alternatively or additionally, excerpts 506 of theindividual reviews may be listed on the same page (e.g., at the bottomend). The list of reviews may be ranked, for instance, based on aranking score computed by the review recommendation system. In someembodiments, the reviews may also be filtered based on criteria selectedby the review consumer. For example, the consumer may be interested onlyin reviews that contradict the findings of the reviewed publication.

FIGS. 6A and 6B show an example user-interface screen showing a detailedindividual review. Again, publication information 600 and/or a link tothe publication being reviewed is displayed, followed by information 602about the reviewer (such as, in addition to her name, scores reflectingthe user's standing within the online social network and/or the impactof her research and publications), including a link to his user-profilepage, and a brief summary 604 of the review (which may, for instance,list the review contributors, the date of the review, the overallconclusion, and the review categories addressed). Underneath, thedetailed answers 606 to the various review questions are shown, andlinks to any supporting documents 608 (e.g., the reviewer's owndatasets) are provided. The review may be followed by viewer comments610 and the reviewer's responses thereto; in some cases, this sectionmay take on the life of a discussion forum related to the publication.The comments may be linked to the commenters' user profiles, orinformation identifying the commenter may otherwise be displayed alongor associated with the comment. Further, to provide readers withinformation that allows them, e.g., to select among large numbers ofcomments, the commenters' reputation may be indicated, e.g., in the formof scores (such as scores computed within and imported from outside thesystem), as shown in FIGS. 6A and 6B. In some embodiments, a separatecolumn includes one or more user-interface elements 612 that solicit anevaluation of the review (e.g., by asking “Was this review helpful toyou?”), summarizes already received feedback (e.g., by indicating howmany review consumers found the review helpful), and/or lists thecommenters themselves.

The social network and publication system described herein can generallybe implemented on a computing system, e.g., in the form of one or moresoftware programs executing on one or more interconnected computers.Users can access the system via a network (such as the Internet), usingany of a number of client devices (such as, e.g., desktop computers,laptop computers, or mobile computing devices including mobiletelephones), to view and/or contribute content (such as publications,reviews, and comments) via appropriate user interfaces. The system maypresent both web interfaces that allow interactions with browser clientapplications, and application programming interfaces (APIs) thatfacilitate interactions with mobile applications executing on mobileclient devices (e.g., smartphones).

In various embodiments, programs and program modules implementing thefunctionality described above include user-interface modules that definethe layout of various screens, facilitate the receipt of user input, andgovern navigation from screen to screen (the “front-end” of the system),as well as logic or processing modules that process data provided to thesystem (e.g., to compute user scores, publication and reviewrecommendations, etc.), manage the storage and organization of the data,and create links between various items of data (the “back-end” of thesystem).

In some embodiments, the data is centrally stored, e.g., on a databaseserver accessible by the various program modules. In other embodiments,the data is stored in a distributed manner, for example, such thatdifferent types of data are located on different storage-media inproximity to the program modules that primarily use them. For example,in one embodiment, the system includes a user-account server that storesuser-account data and user profiles and manages, e.g., the user sign-upprocess as well as the authentication of registered users upon sign-in;a publication server that stores full-text publications and associatedmetadata and provides user interfaces for uploading and viewingpublications, as well as an automatic publication-retrieval module thatscrapes open resources (such as the web) and facilitates bulk uploadsfrom publication repositories; and a review server that stores andmanages reviews and comments and includes program modules implementingthe publication and review recommendation sub-systems (which maydetermine recommendations based at least in part on the reviews andfeedback thereon). Of course, the system's computational functionalityand data can be distributed across multiple servers in many differentways.

FIGS. 7A and 7B illustrate, in block-diagram form, how user data,publication data, and review data may be structured in accordance withan example embodiment. The data generally includes, for each user,associated user-account data 700. The user-account data 700 may includean account identifier (ID) (e.g., an integer number that uniquelyidentifies each account); optionally an indicator of a state of theaccount (e.g., active, inactive, suspended, etc.); login andauthentication information such as a user name and/or login emailaddress and user password; optionally information about the user (e.g.,her full name, contact information, affiliations, etc.); account-historyinformation such as the dates when the account was created or lastmodified, or when the user last logged into the account; accountsettings (specifying, e.g., the user's home page within the system orother user preferences); and/or other types of data. Within the system,a user may act in multiple roles vis a vis a publication, e.g., asauthor of a publication, review requester 704, reviewer 706, reviewcontributor 708, or feedback provider 710 to a review. Regardless whichrole the user assumes in a particular interaction with the system, shewill always be identified with the same (i.e., her unique) user accountID, which may be stored in association with any content she generates.For example, as illustrated, each review request may be stored as a datastructure including the requesting user's account ID; and the metadatafor each review may include the reviewing user's account ID.

Publications may be stored, in accordance with the depicted exampleembodiment, in the form of publication data structure 712 thatrepresents the publication within the social network and publicationsystem, optionally in conjunction with the actual full text of thepublication contained in a separate publication file 714 (e.g., in pdfformat). The publication data 712 may include a publication ID withwhich the publication is uniquely identified within the system; aDigital Object Identifier (DOI) (assigned, e.g., by the social networkand publication system itself for work originally published on thesystem, or by a third-party original publisher for work first publishedexternally to the system) that allows system users to locate thefull-text document if stored externally to the system and/or that allowsexternal Internet users to locate a full-text file 714 stored within thesystem; bibliographic data such as title and author information(including, e.g., the authors' institutional affiliation), an abstractof the publication, the source of the publication (e.g., a particularjournal, if applicable), and the type of the publication (e.g., anarticle, thesis, conference paper, patent, book, book chapter, workingpaper, data set, etc.); the date of posting the publication in thesystem; the account ID of the user who created the publication in thesystem (which may or may not be an author of the publication); and linksto information related to the publication, such as, e.g., reviews. Thefull-text files 714 may be stored within the social network andpublication system itself (e.g., in a separate file repository on thesame database server that also stores the publication data 712, or on aseparate server), or externally to the system (e.g., on a publicallyaccessible web site) such that it can be accessed from within the systemonly via a link. Full-text files 714 may be characterized by a full-textID, the title, file name, file size, mime type, the dates of creationand last modification, which may be stored as part of the file metadata(i.e., within the file itself).

When a user requests a review of the application, a review-request datastructure 716 is created; the review-request data 716 includes a requestID uniquely identifying the request (e.g., such that different usersrequesting reviews for the same publication result in differentinstances of review-request data 716, allowing the system, e.g., totrack how many review requests have been received for a particularpublication) and specifies at least the account ID of the requestinguser and the publication ID for which the review is being requested.This illustrates how various data structures used within the system(such as, e.g., review request 716, publication data 712, anduser-account data 700) are associated with (or “linked to”) each othervia references or cross-reference of their respective IDs.

For each review created, a review data structure 718 is stored in thesystem. The review data structure 718 includes a unique review ID forthe review; an indication of the state of a review (e.g., whether it isin draft, published on the system, pending approval, or rejected); theID of the publication 712 to which the review pertains; the user-accountIDs of the reviewer himself and any contributors to the review; datesassociated with the review, such as the creation date, last-modificationdate, and publication date of the review; the overall conclusion of thereview (e.g., whether it is reproducible, as may be indicatednumerically with a score of 1, −1, or 0 (for not applicable)); and linksto data 720 for the individual review sections and/or files 722associated with the review. The review files 722 may be characterized bya title, file name, file size, mime type, and creation andlast-modification dates.

As described above, a review may include answers to a number of specificquestions. These questions may be grouped into multiple categories (eachof which may include multiple questions, but may, in an extreme case,have only a single question) that correspond to different reviewsections focusing on different aspects of a publication (such asreproducibility, appropriateness of methodology and quality of analysis,importance, etc.). The categories themselves may be defined in areview-category data structure 724 that is identified by a category keyand establishes the sorting priority of the questions within thecategory. The questions themselves as well as the answers provided bythe reviewer may be stored in separate review-section data structures720, one for each category, that are identified by their respectivesection IDs (which are unique across both reviews and categories) andmay specify creation and last-modification dates. The review-sectiondata 720 identifies the associated category by its key and stores thetext of the various question(s) within the category, the reviewer'snumerical answers to these questions (i.e., his ratings), as well as anyfree-text comments she provides on the section. Further, thereview-section data 720 may include references to supporting files 726(herein “section files”) attached by the reviewer (which, like thereview files 722, may be characterized by a title, file name, file size,mime type, and creation and last-modification dates), and pagereferences to portions of the publication upon which the ratings arebased. In some embodiments, page references that link publications andreview sections give rise to separate page-reference data structures728, which specify the ID of and page(s) within the full-text file 714of the publication and the section ID of the review section 720 to whichthey pertain.

Further, as described above, the system may provide readers of thereviews with the ability to rate a review (e.g., in terms of hishelpfulness) and/or provide verbal comments on the review. Thisinformation is stored in review-feedback data structures 728 andreview-comment data structures 730, respectively. The feedback datastructure 728 includes a unique feedback ID by which it can beidentified, the user-account ID of the feedback provider, the review IDof the review to which the feedback pertains, optionally creation andlast-modification dates, and the feedback provider's overall rating ofthe review (e.g., as helpful or not helpful). The review-comment datastructure 730 includes, similarly, a unique review-comment ID, theuser-account ID of the reviewing user and the review ID, optionallycreation and last-modification dates, and the text of the user-providedcomment. In addition, the review-comment data 730 may include flags thatindicate if the comment was made by the author of publication subject tothe review or by the reviewer. A review comment may also link to orreference a separate comment file 732 upload by the commenting user inconnection with the comment.

Accordingly, the overall user, publication, review, and feedback datamay be stored as a combination of structured data (e.g., data structures700, 712, 718, 720, 728, etc.), which may be organized, e.g., in variousdatabase tables, and (unstructured) files (such as, e.g., pdf documents)(e.g., full-text publication files 714 and review/feedback-supportingfiles 722, 726,732).

FIG. 8 illustrates, in block-diagram form, the relations between variousprogram modules 800, data 802, and user interfaces 804 in accordancewith one example embodiment of a social network and publication systemas disclosed herein. As depicted, the system may include a publicationpresentation module 806 that generates, based on stored publication data808 (such as publication metadata structures 712 and/or full-text files714), a publication user interface 810, e.g., as shown in FIG. 2A. (By“generating a user interface” is meant, in this context, the assembly ofall data and layout information used (e.g., by a client device connectedto the system upon transmission thereto) to create a screen display inaccordance herewith (e.g., as depicted in FIG. 2A).) A user-initiatedreview request submitted via a suitable component of the publicationuser interface 810 triggers the solicitation, receipt, and processing ofreview information by a review processing module 812.

The review processing module 812, which is generally a back-endcomponent, may include or, alternatively, call a review solicitationmodule 814 that creates (based, e.g., on the review categories datastructure 724 and/or portions of the publication data 808) thereview-input user interface(s) 816 through which a reviewing user canprovide review information, such as the user interfaces depicted inFIGS. 4A-4C. The user input provided through interface(s) 816 isprocessed by the review processing module 812 to create and/or updatedata objects storing the review data 818 (such as, e.g., review andreview-section data structures 718, 720 and any associated review orsection files 722, 726). A review presentation module 820, in turn,generates a review user interface 822 (such as the user interfacesdepicted in FIGS. 5 and 6A-6B) based on the review data 818. The reviewuser interface 822 may include functionality allowing a review consumerto provide feedback or comments on the review. Such user input islikewise processed by the review processing module 812, resulting in thecreation of comments and feedback data objects 824 (such as, e.g.,review feedback data structures 728, review comment data structures 730,and/or comment files 732). The review presentation module 820 may readin the comments and feedback data 824 to display them along with thereview data or otherwise adjust the review user interface 822 to reflectthe feedback and comments. The review processing module 812 may alsoestablish links or connections between review data 818 and thepublication to which it pertains as well as between comments andfeedback data 824 and the review to which it pertains, e.g., by creatingsuitable entries in a data structure storing reference data 826. Thereference data 826 may be used by both the publication presentationmodule 806 and the review presentation module 820 to update thepublication and review user interfaces 810, 822 with the associatedreviews and comments/feedback or links thereto.

Of course, as will be readily understood by those of ordinary skill inthe art, the depicted modules 808, 812, 820 represent only one amongmany possible ways for organizing the system's review-processing andpublication/review-presentation functionality. In alternativeembodiments, the creation and management of user interfaces and data canbe distributed over fewer, more, and/or different modules than shown inFIG. 8.

Modules, Components, and Logic

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute eithersoftware modules (e.g., code embodied on a machine-readable medium) orhardware modules. A “hardware module” is a tangible unit capable ofperforming certain operations and may be configured or arranged in acertain physical manner. In various example embodiments, one or morecomputer systems (e.g., a standalone computer system, a client computersystem, or a server computer system) or one or more hardware modules ofa computer system (e.g., a processor or a group of processors) may beconfigured by software (e.g., an application or application portion) asa hardware module that operates to perform certain operations asdescribed herein.

In some embodiments, a hardware module may be implemented mechanically,electronically, or using any suitable combination thereof. For instance,a hardware module may include dedicated circuitry or logic that ispermanently configured to perform certain operations. For example, ahardware module may be a special-purpose processor, such as aField-Programmable Gate Array (FPGA) or an Application SpecificIntegrated Circuit (ASIC). A hardware module may also includeprogrammable logic or circuitry that is temporarily configured bysoftware to perform certain operations. For example, a hardware modulemay include software executed by a general-purpose processor or otherprogrammable processor. Once configured by such software, hardwaremodules become specific machines (or specific components of a machine)uniquely tailored to perform the configured functions, and are no longergeneral-purpose processors. It will be appreciated that the decision toimplement a hardware module mechanically, in dedicated and permanentlyconfigured circuitry, or in temporarily configured circuitry (e.g.,configured by software) may be driven by cost and time considerations.

Accordingly, the phrase “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. As used herein,“hardware-implemented module” refers to a hardware module. Consideringembodiments in which hardware modules are temporarily configured (e.g.,programmed), each of the hardware modules need not be configured orinstantiated at any one instance in time. For example, where a hardwaremodule comprises a general-purpose processor configured by software tobecome a special-purpose processor, the general-purpose processor may beconfigured as respectively different special-purpose processors (e.g.,comprising different hardware modules) at different times. Softwareaccordingly configures a particular processor or processors, forexample, to constitute a particular hardware module at one instance oftime and to constitute a different hardware module at a differentinstance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multiplehardware modules exist contemporaneously, communications may be achievedthrough signal transmission (e.g., over appropriate circuits and buses)between or among two or more of the hardware modules. In embodiments inwhich multiple hardware modules are configured or instantiated atdifferent times, communications between such hardware modules may beachieved, for example, through the storage and retrieval of informationin memory structures to which the multiple hardware modules have access.For example, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions describedherein. As used herein, “processor-implemented module” refers to ahardware module implemented using one or more processors.

Similarly, the methods described herein may be at least partiallyprocessor-implemented, with a particular processor or processors beingan example of hardware. For example, at least some of the operations ofa method may be performed by one or more processors orprocessor-implemented modules. Moreover, the one or more processors mayalso operate to support performance of the relevant operations in a“cloud computing” environment or as a “software as a service” (SaaS).For example, at least some of the operations may be performed by a groupof computers (as examples of machines including processors), with theseoperations being accessible via a network (e.g., the Internet) and viaone or more appropriate interfaces (e.g., an Application ProgramInterface (API)).

The performance of certain of the operations may be distributed amongthe processors, not only residing within a single machine, but deployedacross a number of machines. In some example embodiments, the processorsor processor-implemented modules may be located in a single geographiclocation (e.g., within a home environment, an office environment, or aserver farm). In other example embodiments, the processors orprocessor-implemented modules may be distributed across a number ofgeographic locations.

Machine and Software Architecture

The modules, methods, applications and so forth described in conjunctionwith FIGS. 1-8 are implemented in some embodiments in the context of amachine and an associated software architecture. FIGS. 9 and 10 depictrepresentative software and machine (e.g., hardware) architectures thatare suitable for use with the disclosed embodiments. Softwarearchitectures are used in conjunction with hardware architectures tocreate devices and machines tailored to particular purposes.

FIG. 9 is a block diagram 900 illustrating a representative softwarearchitecture 902, which may be used in conjunction with various hardwarearchitectures herein described. FIG. 9 is merely a non-limiting exampleof a software architecture and it will be appreciated that many otherarchitectures may be implemented to facilitate the functionalitydescribed herein. The software architecture 902 may be executing onhardware such as machine 1000 of FIG. 10 that includes, among otherthings, processors 1004, memory 1006, and I/O components 1008. Arepresentative hardware layer 904 is illustrated and can represent, forexample, the machine 900 of FIG. 10. The representative hardware layer904 comprises one or more processing units 906 having associatedexecutable instructions 908. Executable instructions 908 represent theexecutable instructions of the software architecture 902, includingimplementation of the methods, modules and so forth of FIGS. 1-7B.Hardware layer 904 also includes memory and/or storage modules 910,which also have executable instructions 908. Hardware layer 904 may alsocomprise other hardware as indicated by 912.

In the example architecture of FIG. 9, the software 902 may beconceptualized as a stack of layers where each layer provides particularfunctionality. For example, the software 902 may include layers such asan operating system 914, libraries 916, frameworks/middleware 918,applications 920, and presentation layer 922. Operationally, theapplications 920 and/or other components within the layers may invokeapplication programming interface (API) calls 924 through the softwarestack and receive a response, returned values, and so forth illustratedas messages 926 in response to the API calls 924. The layers illustratedare representative in nature and not all software architectures have alllayers. Other software architectures may include additional or differentlayers.

The operating system 914 may manage hardware resources and providecommon services. The operating system 914 may include, for example, akernel 928, services 930, and drivers 932. The kernel 928 may act as anabstraction layer between the hardware and the other software layers.For example, the kernel 928 may be responsible for memory management,processor management (e.g., scheduling), component management,networking, security settings, and so on. The services 930 may provideother common services for the other software layers. The drivers 932 maybe responsible for controlling or interfacing with the underlyinghardware. For instance, the drivers 932 may include display drivers,camera drivers, Bluetooth® drivers, flash memory drivers, serialcommunication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi®drivers, audio drivers, power management drivers, and so forth dependingon the hardware configuration.

The libraries 916 may provide a common infrastructure that may beutilized by the applications 920 and/or other components and/or layers.The libraries 916 typically provide functionality that allows othersoftware modules to perform tasks in an easier fashion than to interfacedirectly with the underlying operating system 914 functionality (e.g.,kernel 928, services 930 and/or drivers 932). The libraries 916 mayinclude system 934 libraries (e.g., C standard library) that may providefunctions such as memory allocation functions, string manipulationfunctions, mathematic functions, and the like. In addition, thelibraries 916 may include API libraries 936 such as media libraries(e.g., libraries to support presentation and manipulation of variousmedia format such as MPREG4, H.264, MP3, AAC, AMR, JPG, PNG), graphicslibraries (e.g., an OpenGL framework that may be used to render 2D and3D in a graphic content on a display), database libraries (e.g., SQLitethat may provide various relational database functions), web libraries(e.g., WebKit that may provide web browsing functionality), and thelike. The libraries 916 may also include a wide variety of otherlibraries 938 to provide many other APIs to the applications 920 andother software components/modules.

The frameworks 918 (also sometimes referred to as middleware) mayprovide a higher-level common infrastructure that may be utilized by theapplications 920 and/or other software components/modules. For example,the frameworks 918 may provide various graphic user interface (GUI)functions, high-level resource management, high-level location services,and so forth. The frameworks 918 may provide a broad spectrum of otherAPIs that may be utilized by the applications 920 and/or other softwarecomponents/modules, some of which may be specific to a particularoperating system or platform.

The applications 920 may utilize built-in operating system functions(e.g., kernel 928, services 930 and/or drivers 932), libraries (e.g.,system 934, APIs 936, and other libraries 938), frameworks/middleware918 to create user interfaces to interact with users of the system.Alternatively, or additionally, in some systems interactions with a usermay occur through a presentation layer, such as presentation layer 922.In these systems, the application/module “logic” can be separated fromthe aspects of the application/module that interact with a user.

Some software architectures utilize virtual machines. In the example ofFIG. 9, this is illustrated by virtual machine 948. A virtual machinecreates a software environment where applications/modules can execute asif they were executing on a hardware machine (such as the machine ofFIG. 10, for example). A virtual machine is hosted by a host operatingsystem and typically, although not always, has a virtual machine monitor946, which manages the operation of the virtual machine as well as theinterface with the host operating system (i.e., operating system 914). Asoftware architecture executes within the virtual machine such as anoperating system 950, libraries 952, frameworks/middleware 954,applications 956 and/or presentation layer 958. These layers of softwarearchitecture executing within the virtual machine 948 can be the same ascorresponding layers previously described or may be different.

FIG. 10 is a block diagram illustrating components of a machine 900,according to some example embodiments, able to read instructions from amachine-readable medium (e.g., a machine-readable storage medium) andperform any one or more of the methodologies discussed herein.Specifically, FIG. 10 shows a diagrammatic representation of the machine1000 in the example form of a computer system within which instructions1002 (e.g., software, a program, an application, an applet, an app, orother executable code) for causing the machine 1000 to perform any oneor more of the methodologies discussed herein may be executed. Forexample, the instructions may cause the machine to execute the flowdiagrams of FIGS. 3A-3B, and/or may implement the publicationpresentation, review processing, and review presentation modules 806,812, 820 of FIG. 8. The instructions 1002 transform the general,non-programmed machine into a particular machine programmed to carry outthe described and illustrated functions in the manner described. Inalternative embodiments, the machine 1000 operates as a standalonedevice or may be coupled (e.g., networked) to other machines. In anetworked deployment, the machine 1000 may operate in the capacity of aserver machine or a client machine in a server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine 1000 may comprise, but not be limitedto, a server computer, a client computer, a personal computer (PC), atablet computer, a laptop computer, a netbook, a set-top box (STB), apersonal digital assistant (PDA), a mobile device, a web appliance, orany machine capable of executing the instructions 1002, sequentially orotherwise, that specify actions to be taken by machine 1000. Further,while only a single machine 1000 is illustrated, the term “machine”shall also be taken to include a collection of machines 1000 thatindividually or jointly execute the instructions 1002 to perform any oneor more of the methodologies discussed herein.

The machine 1000 may include processors 1004, memory 1006, and I/Ocomponents 1008, which may be configured to communicate with each othersuch as via a bus 1010. In an example embodiment, the processors 1004(e.g., a Central Processing Unit (CPU), a Reduced Instruction SetComputing (RISC) processor, a Complex Instruction Set Computing (CISC)processor, a Graphics Processing Unit (GPU), a Digital Signal Processor(DSP), an Application Specific Integrated Circuit (ASIC), aRadio-Frequency Integrated Circuit (RFIC), another processor, or anysuitable combination thereof) may include, for example, processor 1012and processor 1014 that may execute instructions 1002. The term“processor” is intended to include multi-core processor that maycomprise two or more independent processors (sometimes referred to as“cores”) that may execute instructions contemporaneously. Although FIG.10 shows multiple processors, the machine 1000 may include a singleprocessor with a single core, a single processor with multiple cores(e.g., a multi-core process), multiple processors with a single core,multiple processors with multiples cores, or any combination thereof.

The memory/storage 1006 may include a memory 1016, such as a mainmemory, or other memory storage, and a storage unit 1018, bothaccessible to the processors 1004 such as via the bus 1010. The storageunit 1018 and memory 1016 store the instructions 1002 embodying any oneor more of the methodologies or functions described herein. Theinstructions 1002 may also reside, completely or partially, within thememory 1016, within the storage unit 1018, within at least one of theprocessors 1004 (e.g., within the processor's cache memory), or anysuitable combination thereof, during execution thereof by the machine1000. Accordingly, the memory 1016, the storage unit 1018, and thememory of processors 1004 are examples of machine-readable media.

As used herein, “machine-readable medium” means a device able to storeinstructions and data temporarily or permanently and may include, but isnot be limited to, random-access memory (RAM), read-only memory (ROM),buffer memory, flash memory, optical media, magnetic media, cachememory, other types of storage (e.g., Erasable Programmable Read-OnlyMemory (EEPROM)) and/or any suitable combination thereof. The term“machine-readable medium” should be taken to include a single medium ormultiple media (e.g., a centralized or distributed database, orassociated caches and servers) able to store instructions 1002. The term“machine-readable medium” shall also be taken to include any medium, orcombination of multiple media, that is capable of storing instructions(e.g., instructions 1002) for execution by a machine (e.g., machine1000), such that the instructions, when executed by one or moreprocessors of the machine 1000 (e.g., processors 1004), cause themachine 1000 to perform any one or more of the methodologies describedherein. Accordingly, a “machine-readable medium” refers to a singlestorage apparatus or device, as well as “cloud-based” storage systems orstorage networks that include multiple storage apparatus or devices. Theterm “machine-readable medium” excludes signals per se.

The I/O components 1008 may include a wide variety of components toreceive input, provide output, produce output, transmit information,exchange information, and so on. The specific I/O components 1008 thatare included in a particular machine will depend on the type of machine.For example, portable machines such as mobile phones will likely includea touch input device or other such input mechanisms, while a headlessserver machine will likely not include such a touch input device. Itwill be appreciated that the I/O components 1008 may include many othercomponents that are not shown in FIG. 10. The I/O components 1008 aregrouped according to functionality merely for simplifying the followingdiscussion and the grouping is in no way limiting. In various exampleembodiments, the I/O components 1008 may include output components 1020and input components 1022. The output components 1020 may include visualcomponents (e.g., a display such as a plasma display panel (PDP), alight emitting diode (LED) display, a liquid crystal display (LCD), aprojector, or a cathode ray tube (CRT)), acoustic components (e.g.,speakers), haptic components (e.g., a vibratory motor, resistancemechanisms), other signal generators, and so forth. The input components1022 may include alphanumeric input components (e.g., a keyboard, atouch screen configured to receive alphanumeric input, a photo-opticalkeyboard, or other alphanumeric input components), point-based inputcomponents (e.g., a mouse, a touchpad, a trackball, a joystick, a motionsensor, or other pointing instrument), tactile input components (e.g., aphysical button, a touch screen that provides location and/or force oftouches or touch gestures, or other tactile input components), audioinput components (e.g., a microphone), and the like.

Communication may be implemented using a wide variety of technologies.The I/O components 1008 may include communication components 1024operable to couple the machine 1000 to a network 1026 or devices 1030via coupling 1032 and coupling 1034, respectively. For example, thecommunication components 1024 may include a network interface componentor other suitable device to interface with the network 1026. In furtherexamples, communication components 1024 may include wired communicationcomponents, wireless communication components, cellular communicationcomponents, Near Field Communication (NFC) components, Bluetooth®components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and othercommunication components to provide communication via other modalities.The devices 1030 may be another machine or any of a wide variety ofperipheral devices (e.g., a peripheral device coupled via a UniversalSerial Bus (USB)).

A variety of information may be derived via the communication components1024, such as, location via Internet Protocol (IP) geo-location,location via Wi-Fi® signal triangulation, location via detecting a NFCbeacon signal that may indicate a particular location, and so forth.

In various example embodiments, one or more portions of the network 1026may be an ad hoc network, an intranet, an extranet, a virtual privatenetwork (VPN), a local area network (LAN), a wireless LAN (WLAN), a widearea network (WAN), a wireless WAN (WWAN), a metropolitan area network(MAN), the Internet, a portion of the Internet, a portion of the PublicSwitched Telephone Network (PSTN), a plain old telephone service (POTS)network, a cellular telephone network, a wireless network, a Wi-Fi®network, another type of network, or a combination of two or more suchnetworks. For example, the network 1026 or a portion of the network 1026may include a wireless or cellular network and the coupling 1032 may bea Code Division Multiple Access (CDMA) connection, a Global System forMobile communications (GSM) connection, or other type of cellular orwireless coupling. In this example, the coupling 1032 may implement anyof a variety of types of data transfer technology, such as SingleCarrier Radio Transmission Technology (1×RTT), Evolution-Data Optimized(EVDO) technology, General Packet Radio Service (GPRS) technology,Enhanced Data rates for GSM Evolution (EDGE) technology, thirdGeneration Partnership Project (3GPP) including 3G, fourth generationwireless (4G) networks, Universal Mobile Telecommunications System(UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability forMicrowave Access (WiMAX), Long Term Evolution (LTE) standard, othersdefined by various standard setting organizations, other long rangeprotocols, or other data transfer technology.

The instructions 1002 may be transmitted or received over the network1026 using a transmission medium via a network interface device (e.g., anetwork interface component included in the communication components1024) and utilizing any one of a number of well-known transfer protocols(e.g., hypertext transfer protocol (HTTP)). Similarly, the instructions1002 may be transmitted or received using a transmission medium via thecoupling 1034 (e.g., a peer-to-peer coupling) to devices 1030. The term“transmission medium” shall be taken to include any intangible mediumthat is capable of storing, encoding, or carrying instructions 1002 forexecution by the machine 1000, and includes digital or analogcommunications signals or other intangible medium to facilitatecommunication of such software.

Language

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Although an overview of the inventive subject matter has been describedwith reference to specific example embodiments, various modificationsand changes may be made to these embodiments without departing from thebroader scope of embodiments of the present disclosure.

The embodiments illustrated herein are described in sufficient detail toenable those skilled in the art to practice the teachings disclosed.Other embodiments may be used and derived therefrom, such thatstructural and logical substitutions and changes may be made withoutdeparting from the scope of this disclosure. The Detailed Description,therefore, is not to be taken in a limiting sense, and the scope ofvarious embodiments is defined only by the appended claims, along withthe full range of equivalents to which such claims are entitled.

As used herein, the term “or” may be construed in either an inclusive orexclusive sense. Moreover, plural instances may be provided forresources, operations, or structures described herein as a singleinstance. Additionally, boundaries between various resources,operations, modules, engines, and data stores are somewhat arbitrary,and particular operations are illustrated in a context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within a scope of various embodiments of thepresent disclosure. In general, structures and functionality presentedas separate resources in the example configurations may be implementedas a combined structure or resource. Similarly, structures andfunctionality presented as a single resource may be implemented asseparate resources. These and other variations, modifications,additions, and improvements fall within a scope of embodiments of thepresent disclosure as represented by the appended claims. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense.

What is claimed is:
 1. Within an online social network and publicationsystem having a plurality of registered users, a method comprising, byone or more processors of the system executing instructions stored inone or more computer-readable media: for each of a plurality of researchpublications, generating a respective publication user interface,accessible by each of the registered users, that displays or links tothe research publication, the publication user interface comprising oneor more user-interface elements enabling each of the registered users toinitiate a review by issuing a digital request to review the researchpublication and further enabling each of the registered users toinitiate a review by issuing a digital request for solicitation of areview of the research publication from one or more other users that areto be automatically determined by the system without input from the userrequesting solicitation of the review; in response to receipt of adigital request for solicitation of a review from one or more otherusers, automatically identifying, among the registered users, one ormore that are researchers in a field of the respective researchpublication based at least in part on a level of interest-relatednesswith the research publication, and inviting the one or more identifiedregistered users to become reviewers; and upon acceptance, by one of theidentified registered users, of an invitation to become a reviewer, orin response to receipt, from any of the registered users, of a digitalrequest to review the research publication, soliciting and receivingdigital review information from the respective identified or requestingregistered user via a review user interface, adjusting impact of thedigital review information in consideration of a level of biases of thereviewer based on the level of social relatedness of the reviewer withan author of the research publication, and updating the publication userinterface with at least a portion of the digital review information or alink to at least a portion of the digital review information; and makingpublication and review recommendations to the registered users based atleast in part on an analysis of a level of social relatedness betweenthe registered users and a level of interest relatedness between theregistered users and the research publications and digital reviewinformation.
 2. The method of claim 1, wherein the one or more usersthat are researchers in the field of the respective research publicationare identified based on at least one of digital user profiles associatedwith the one or more users or a relatedness of the one or more user'sinterests to a topic of the publication, the relatedness determined bythe social network and publication system.
 3. The method of claim 2,wherein the relatedness of the topic of the publication to interests ofa user to be invited to become reviewing user is determined based on acluster of topics computed from the user's own publications.
 4. Themethod of claim 1, wherein the digital review information comprisesanswers to one or more questions relating to reproducibility or qualityof the research described in the research publication and one or morefree-text comments supporting the one or more answers.
 5. The method ofclaim 4, wherein the digital review information further comprisessupporting data associated with one or more of the answers.
 6. Themethod of claim 5, further comprising associating the supporting datawith a digital user profile associated with the reviewing user.
 7. Themethod of claim 4, wherein soliciting and receiving digital reviewinformation comprises allowing the reviewing user to select a portion ofthe publication for insertion into the free-text comment, andautomatically supplementing the free-text comment with a reference tothe selected portion.
 8. The method of claim 7, further comprisingautomatically annotating the referenced portion of the publication,based on the reference thereto in the free-text comment, with theassociated answer or a link thereto.
 9. The method of claim 7, whereinallowing the reviewing user to select a portion of the publication forinsertion into the free-text comment comprises displaying auser-interface component that allows the user to manually select theportion of the publication.
 10. The method of claim 4, wherein thequestions further comprise one or more questions related to at least oneof appropriateness of a methodology or analysis applied, inclusion ofrelevant citations to related work, novelty and/or importance ofresearch findings, or justification of a conclusion in view of researchresults.
 11. The method of claim 1, further comprising receiving digitalreview information from multiple reviewing users, automaticallyaggregating the received digital review information, and updating thepublication user interface with the aggregated digital reviewinformation or a link to the aggregated digital user information. 12.The method of claim 11, wherein soliciting the review informationcomprises presenting the reviewing user with a predetermined set of oneor more questions relating to multiple aspects of research-publicationquality and an associated predetermined set of answer choices for eachof the questions, and wherein automatically aggregating the reviewinformation comprises counting, for each of the predetermined questionsand each of a number of available answer choices for the question, anumber of times that answer choice has been selected by the multiplereviewing users.
 13. The method of claim 12, wherein the publicationuser interface is updated with a review overview, or a link to a reviewoverview, generated based on the aggregated digital review information,the review overview comprising, for each of the questions, the number oftimes each of the answer choices has been selected by the multiplereviewing users, and further comprising one or more links to one or moreuser interfaces including the review information received from themultiple reviewing users individually.
 14. The method of claim 1,further comprising receiving, via a user interface element provided on auser interface including at least a portion of the review information,user feedback related to the review information.
 15. The method of claim14, wherein the feedback comprises an indication of a selected level ofhelpfulness of the review information.
 16. The method of claim 14,further comprising, following receipt of the feedback, providing accessto the feedback on the user interface including at least a portion ofthe review information.
 17. The method of claim 16, further comprisingindicating, on the user interface providing access to the feedback, ifthe feedback originates from an author of the publication or from thereviewing user.
 18. The method of claim 16, further comprisingdisplaying, in association with providing access to the feedback, anidentity of a feedback-providing user and information indicative of thefeedback-providing user's reputation.
 19. The method of claim 14,further comprising processing the feedback and, based thereon, adjustingat least one of a score associated with the reviewing user or a rank ofa set of review information received from one reviewing user among aplurality of sets of review information received for the publicationfrom multiple reviewing users.
 20. The method of claim 1, wherein thereview information comprises a list of one or more people other than thereviewer that have contributed to the review.
 21. The method of claim 1,further comprising displaying an identity of the reviewing user andinformation indicative of the reviewing user's reputation in associationwith the review information.
 22. The method of claim 1, furthercomprising enabling each of the registered users to post researchpublications that the respective user authored to the system or to claimauthorship of research publications already available on the system. 23.A social network and publication system comprising: one or moreprocessors; and one or more machine-readable media configured to storedata for each of a plurality of research publications and for each of aplurality of reviews, each review being associated with one of theplurality of research publications, and instructions which, whenexecuted by the one or more processors, cause the one or more processorsto perform operations comprising: for each of the plurality of researchpublications, generating a respective publication user interface,accessible by each of a plurality of registered users of the socialnetwork and publication system, that displays or links to the researchpublication, the publication user interface comprising one or moreuser-interface elements enabling each of the registered users toinitiate a review by issuing a digital request to review the researchpublication and further enabling each of the registered users toinitiate a review by issuing a digital request for solicitation of areview of the research publication from one or more other users that areto be automatically determined by the system without input from the userrequesting solicitation of the review; in response to receipt of adigital request for solicitation of a review from one or more otherusers, automatically identifying, among the registered users, one ormore users that are researchers in a field of the respective researchpublication based at least in part on a level of interest-relatednesswith the research publication, and inviting the one or more identifiedregistered users to become reviewers; and upon acceptance, by one of theidentified registered users, of an invitation to become a reviewer, orin response to receipt, from any of the registered users, of a digitalrequest to review the research publication, soliciting and receivingdigital review information from the respective identified or requestingregistered user via a review user interface, adjusting impact of thedigital review information in consideration of a level of biases of thereviewer based on the level of social relatedness of the reviewer withan author of the research publication, and updating the respectivepublication user interface with at least a portion of the digital reviewinformation or a link to at least a portion of the digital reviewinformation; and making publication and review recommendations to theregistered users based at least in part on an analysis of a level ofsocial relatedness between the registered users and a level of interestrelatedness between the registered users and the research publicationsand digital review information.